Dual coding of knowledge in the human brain

Neural systems have been assumed to represent knowledge through the embodiment of sensory experiences. This type representation is considered fundamentally different from artificial intelligence approach, which implements using ‘symbolic’ approach.Recent behavioral and neuroimaging studies color (and other visual knowledge) with without experience provide empirical evidence for neural coding non-sensory, language-derived knowledge, along sensory-derived representation, in brain systems.The proposed dual-coding framework motivated by these recent findings offers nature’s solution classical challenges associated either only symbolic tradition or embodied approach neuroscience. How does human code about world? While disciplines such as world based on language, neurocognitive models dominated embodiment, derived sensory/motor supported high-level association cortices. The correlates an alternative disembodied system had previously difficult establish. A line exploring properties, color, visually deprived individuals converge positive, compelling language-derived, dorsal anterior temporal lobe extended language network, addition representations, leading a sketch framework. (A android): There was thought experiment they gave us. It’s called ‘Mary Black White room.’ Mary scientist, her special subject color. She knows everything there know it […] But she lives black white room. born there, raised One day… walks out sees blue sky. And at that moment learns something all couldn’t tell her. what feels see show students difference between computer mind. when out. – Ex Machina (a Movie, 2014), citing [1.Jackson F. Epiphenomenal qualia.Philos. Q. 1982; 32: 127-136Crossref Google Scholar]. What mean, know? Is knowing relations versus (see Glossary) (current) machines humans, suggested earlier movie dialogue? stores vast amounts world, roses, shape earth, mechanisms evolution, functions jury. Various sorts guide broad range behaviors, recognizing objects understanding words, reasoning, decision making. Understanding how represented one oldest most-debated questions any discipline interested mind brain. discussions every field, including philosophy, psychology, cognitive science, center two notions representing knowledge: (sensory/motor-experience-derived). granted room Scholar], prevalent way intelligence. Beyond computers, similar also various (e.g., dual both verbal nonverbal [2.Paivio A. Mental Representations: Dual Coding Approach. Oxford University Press, 1986Google Scholar,3.Paivio Mind Its Evolution: Theoretical Lawrence Erlbaum Associates Publishers, 2007Google Scholar]) philosophical theories dualism [4.Papineau D. Thinking About Consciousness. 2002Crossref Scholar]). Despite this popularity some areas, (biological) implementation are more strongly influenced notion ‘sensory embodiment’, start assumptions least physical apple), acquired grounded experiences stored higher-order cortices [5.Binder J.R. Desai R.H. neurobiology semantic memory.Trends Cogn. Sci. 2011; 15: 527-536Abstract Full Text PDF PubMed Scopus (1067) Scholar, 6.Martin GRAPES—grounding representations action, perception, emotion systems: object properties categories brain.Psychon. Bull. Rev. 2016; 23: 979-990Crossref (140) 7.Pulvermüller neurons make meaning: Brain abstract-symbolic semantics.Trends 2013; 17: 458-470Abstract (305) 8.Barsalou L.W. On staying avoiding quixotic dead ends.Psychon. 1122-1142Crossref (128) 9.Lambon Ralph M.A. et al.The computational bases cognition.Nat. Neurosci. 2017; 18: 42-55Crossref (650) 10.Patterson K. al.Where do you brain.Nat. 2007; 8: 976-987Crossref (1690) 11.Binder In defense abstract conceptual representations.Psychon. 1096-1108Crossref (96) 12.Barsalou Grounded cognition.Annu. Psychol. 2008; 59: 617-645Crossref (3619) review, I discuss identifies non-sensory form representation. To explain rationales help readers appreciate their implications first briefly present current views embodiment’ stems combination considerations: biological entities acquire information interaction world; long identified brain; numerous during over decades shown retrieving activates regions territory corresponding (primary associative) For instance, roses) induces activation close V4, region critical perception [13.Martin al.Discrete cortical action.Science. 1995; 270: 102-105Crossref (833) Scholar,14.Simmons W.K. al.A common substrate perceiving color.Neuropsychologia. 45: 2802-2810Crossref (280) processing words whose meanings strong associations tends modulate activities [15.Fernandino L. al.Concept reflects multimodal abstraction: semantics.Cereb. Cortex. 26: 2018-2034Crossref (125) Thus, often assume, extents, can be perceived modalities, abstracted experiences/systems connections set combinational It axiomatic humans learn via things not personally experienced sensorily someone ‘roses red’ reading/hearing seeing red roses), yet, quoting [16.Binder al.Toward brain-based componential representation.Cogn. Neuropsychol. 33: 130-174Crossref (126) Scholar] ‘In cognition theories, however, our model maintains concepts ultimately experience, whether directly indirectly refers experience’. research has away specific given [17.Mahon B.Z. Caramazza look hypothesis new proposal grounding content.J. Physiol. Paris. 102: 59-70Crossref (841) Scholar]), isolate, articulate, identify supporting non-sensory-derived knowledge. because typically developed brain, evoked even justice) cannot perceive quark). cases deprivation input modality, congenital blindness, most still gained modalities apple obtained touch vision). difficulty cognitively excluding leads further mapping activity cognition. clean test would involve testing happens particular modality absent. exactly why philosophers Scholar,18.Locke J. An Essay Concerning Human Understanding. 1690Crossref fascinated Color sensation exclusively visual, driven wavelength light, no senses it. Colors congenitally blind individual mirror What, how, brains sight colors visual-exclusive knowledge)? Answers opportunity isolate following sections, will review provided answers compelling, positive existence highly sensory-related wave empirically started almost four ago. showed people types reviewed [19.Bedny M. Saxe R. Insights into origins neuroscience blindness.Cogn. 2012; 29: 56-84Crossref (31) Briefly, early work found children use adjectives names speech, perceptual qualities felt touch, name [20.Landau B. Gleitman Language Experience: Evidence Blind Child. Harvard 1985Google 21.Connolly A.C. al.Effect blindness everyday concepts.Proc. Natl. Acad. U. S. 104: 8241-8246Crossref (42) 22.Mills A.E. Acquisition Child: Normal Deficient. College-Hill 1983Google group free-recall tasks [23.Zimler Keenan J.M. Imagery blind: images?.J. Exp. Learn. Mem. 1983; 9: 269-282Crossref (91) recover Newtonian circle labels [24.Shepard R.N. Cooper L.A. Representation blind, color-blind, normally sighted.Psychol. 1992; 3: 97-104Crossref (123) Scholar,25.Marmor G.S. Age onset development semantics names.J. Child 1978; 25: 267-278Crossref (44) Given work’s relevance larger question recently rigorously investigated, detailed benchmark comparisons, detail below. comparison sighted individuals, multiple (basic descriptive terms) tested similarity rating task paired terms (congenital noncongenital) control conditions [26.Saysani al.Colour envisioned: colour sighted.Vis. 2018; 382-392Crossref (15) space overall strikingly (Figure 1A ), subjects indistinguishable sighted, greater variation group. high degree correlation (r = 0.93) term spaces replicated, relationship lower than results action 0.99) [27.Bottini al.Brain involved retrieval people.J. 2020; 1009-1025Crossref (8) Compared intriguing cross-modal reported (red heavier yellow), color-weight associations, but object-taste (boulder-sour, strength benchmarks), suggesting contributes [28.Barilari al.Is yellow blind?.Iperception. 1-4Google Another colors. individuals’ fruits vegetables probed [29.Wang X. al.Two forms brain.Neuron. 107: 383-393Abstract (22) again yielded 1B) relative individuals. Furthermore, strengths typical (nonvisual) (including shape, taste), well general (both continuously rated categorically measured), were groups, indicating differently relying derive relations. investigation carried visual-specific [30.Bedny al.There’s “sparkle” meets eye: vision light verbs among individuals.Cognition. 2019; 189: 105-115Crossref Scholar,31.Bedny al.Growing up change theory mind.Proc. 2009; 106: 11312-11317Crossref (68) peek feel/perceive; shimmer boom/roll), controls (by means word judgment). Multidimensional scaling analyses distinguish, like emission sound 1C). suggests concepts, blindness. Over dozen spanning decades, across paradigms (generation, sorting, rating, association) languages (English, Italian, Mandarin Chinese) paint consistent picture complete deprivation. (i) Without actually sensorially experiencing relational structures, patterns those sighted; (ii) level, non-visual-specific [32.Kim J.S. al.Knowledge animal appearance adults.Proc. 166: 11213-11222Crossref (25) Scholar]); (iii) within population fully quite different, additional may reduce idiosyncrasy [33.Wang, Bi, Y. Idiosyncratic tower babel: Individual differences meaning increase abstractness. (accepted) (bioRxiv preprint, doi: https://doi.org/10.1101/2020.08.28.272997)Google point property absence ways perfectly probing Turing scenario [34.Turing A.M. Computing machinery intelligence.Mind. 1950; 49: 433-460Crossref basis visual-concept vision? Recent fMRI groups important clues, described below 2). responses examined depicting dominant phenomena, ‘red’, ‘rainbow’, ‘island’), features (abstract e.g., ‘freedom’), non-visual accessible ‘square’, ‘rain’, ‘cup’) [35.Striem-Amit E. al.Neural blind.Nat. Commun. 5250Crossref (24) cluster (ATL), superior gyrus/sulcus (STG/STS) pole (TP), significant interaction, stronger ‘rainbow’ (or ‘red’) compared ‘rain’ ‘square’) (‘rainbow’/’red’ ‘rain’/’square’ is), (all accessible; Figure 2 another study adaptation paradigm, judgments produced repetition suppression effects V4 STG (along precentral cortex) group, peak portion These convergently ATL sensitive uncovered representational analysis (RSA) [36.Kriegeskorte N. al.Representational - connecting branches neuroscience.Front. Syst. 2: 1-28PubMed Participants listened retrieved matrix (correlation pattern items) computed correlations taken Two 2): posterior fusiform color-perception-sensitive (close V4) similarity; left (anterior TP), object-color after controlled for. sets intrinsically, functionally connected Note under lenient threshold, ventral/medial observed showing convergence related [9.Lambon Scholar,35.Striem-Amit Although handful addressed question, appear clear. (univariate contrasts, adaptation, RSA) (Italian, (for concepts) while confirming cortex sighted. Importantly, RSA revealed occurs available representations. Are tested, reflect mechanism ATL? studying (i.e., referents, ‘justice’ ‘rationality’). imaging consistently concrete ‘cup’), requiring facts facts, elicit ATL, centering around dorsal-lateral aspects, encompassing network [37.Fedorenko al.New method investigations language: defining ROIs subjects.J. Neurophysiol. 2010; 1177-1194Crossref (276) inferior frontal gyrus (IFG) middle (pMTG) [38.Wang concepts: meta-analysis studies.Hum. Mapp. 31: 1459-1468Crossref (253) 39.Wang al.Close yet independent: dissociation social valence dimensions lobe.Hum. 40: 4759-4776Crossref (19) 40.Hoffman P. al.Differing contributions prefrontal knowledge.Cortex. 2015; 63: 250-266Crossref (115) 41.Binder system? 120 functional studies.Cereb. 19: 2767-2796Crossref (2414) 42.Sabsevitz D.S. al.Modulation imageability.Neuroimage. 2005; 27: 188-200Crossref (236) interpretations controversial researchers debated relate involvement Scholar,42.Sabsevitz emotional [43.Zahn al.Social cortex.Proc. 6430-6435Crossref (344) Scholar,44.Mellem M.S. al.Sentence shows social-emotional bias.Neuropsychologia. 89: 217-224Crossref (30) argued (external and/or internal) due nuanced lexical confounds age acquisition familiarity. observations parsimoniously explained potentially its network) Intriguingly, if perceptible properties? might richly distributed thus diluted network. earlier, primarily focusing jointly considering allows us 3, Key figure ). well-establ